Success Index: Measuring the efficiency of search engines using implicit user feedback”, PCI 2007, Web Search and Mining Track

The problem of characterizing web content and evaluating search engine results in terms of relevance to the user’s intention has been an important issue in web research. Despite the large number of manually categorized web query datasets, which can be used for testing and tuning ranking algorithms, the evaluation problem remains unsolved because of the size of the web, the diversity of the query space, and the subjective nature of user satisfaction. In this paper we study Success Index (SI), a method for evaluating the quality of ranking algorithms. Success Index takes into account a user’s clickthrough data, and provides an evaluation of ranking. Through extensive user blind tests we show that the results of Success Index compare favorably to those of an explicit evaluation. We also review other existing implicit evaluation techniques and summarize the features that can be exploited for evaluating ranking